Papers with evaluation benchmark
CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in foundation language models have shown the efficacy of pre-trained models across diverse QA tasks. |
| Approach: | They propose a multi-task benchmark for evaluating causality-aware language models to unify causal QA research. |
| Outcome: | The proposed model outperforms single-task fine-tuned models on the CALM-Bench tasks. |
Nibbling at the Hard Core of Word Sense Disambiguation (2022.acl-long)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a task that is based on a set of pre-trained language models. |
| Approach: | They propose to use Word Sense Disambiguation to test whether systems can handle ambiguous words. |
| Outcome: | The proposed benchmarks show that seven of the most representative state-of-the-art systems make trivial errors on traditional evaluation benchmarks. |
Towards standardizing Korean Grammatical Error Correction: Datasets and Annotation (2023.acl-long)
Copied to clipboard
| Challenge: | Despite the growing number of Korean learners, little research has been conducted on Korean grammatical error correction (GEC) despite the difficulties of the Korean language, there is no evaluation benchmark for Korean GEC. |
| Approach: | They propose to use Korean grammar error correction datasets to train a machine learning model that can automatically annotate Korean errors from parallel corpora. |
| Outcome: | The proposed model outperforms the currently used statistical Korean GEC system on a wider range of error types. |
CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks such as revising source code to address code reviews. |
| Approach: | They propose a benchmark to evaluate large language models' ability to bridge both technical and conversational contexts by decomposing the generation task of code refinement into three essential reasoning steps. |
| Outcome: | The proposed benchmark exposes specific model weaknesses in code review comprehension disentangled from their generative automated code refinement results. |
Robustness Evaluation of Entity Disambiguation Using Prior Probes: the Case of Entity Overshadowing (2021.emnlp-main)
Copied to clipboard
| Challenge: | Entity disambiguation (ED) is the last step of entity linking when candidate entities are reranked according to the context they appear in. |
| Approach: | They propose a dataset that includes 16K short text snippets annotated with entity mentions to evaluate EL models. |
| Outcome: | The proposed dataset shows that the performance of EL systems is overestimated . the results show that the EL system performance is significantly better on the ShadowLink benchmark . |
AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark (2025.acl-long)
Copied to clipboard
| Challenge: | Evaluation benchmarks based on predefined domains and human-labeled data face limitations in addressing evaluation needs for emerging domains. |
| Approach: | They propose an automated information retrieval benchmark based on predefined domains and human-labeled data . AIR-Bench is automated and Heterogeneous with three key features . |
| Outcome: | The proposed benchmarks are based on predefined domains and human-labeled data. |